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ABSTRACT 



Two instruments were developed to elicit information about 
students’ abilities and their instructional needs in finding information in a 
library. Tests of student skills were developed through a series of empirical 
trials and reviews. The third trial of the test was carried out by using 
whole classes selected randomly to yield two classes from four schools for 
each test. Results show that the developed elementary (grades 5 and 6) and 
intermediate (grades 7 and 8) tests are sufficiently reliable to proceed to 
national standardization. The tests for secondary school students has many 
difficult items, and will require further developmental trials before 
national standardization. Test results show that girls score higher and are 
rated higher than boys, and that students in high-performance schools ("high 
decile") score better than those at low and middle levels of performance. 
Older students also tend to do better than younger ones. The tests seem to be 
moving in the right direction in terms of their ability to discriminate 
reliably among candidates, and they should permit teachers to make informed 
decisions about the learning needs of a wide variety of students. (Contains 
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Assessing an essential skill: 
Finding information in the library 



Success in information literacy partly requires accurate and independent ability to 
locate items in libraries. This paper reports on the author’s exploratory work in 
developing school-based assessments in library skills. Direct tests of student 
knowledge and teacher observational ratings of student independence have been 
developed. The paper will describe how library skills have been defined and 
sequenced for primary (Y5-6), intermediate (Y7-8), and secondary (Y9-10) stages. 
Findings on the reliability and validity of the assessments at all levels will be 
discussed. Specific questions to be addressed include: 

• What role do gender and decile play in test performance and teacher 
observation? 

• What differences are there in achievement in understanding catalogues and 
locating items in a library? 

• What agreement is there in library skills achievement as measured by test 
scores and by teacher observation? 
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In a paper (Brown, 1997) given at the 1997 NZARE conference I discussed 
information skills in the New Zealand curriculum framework and foreshadowed some 
types of assessment instruments that could be developed to assist classroom teachers in 
teaching information skills. This paper follows the development of two instruments 
designed to elicit information about students’ abilities and their instructional needs in 
finding information in a library. These two instruments are part of a collection of direct 
tests, teacher rating schedules, and student self-reports (Table 1) that are being developed 
or planned for phased publication in 1999. Three levels of the test are in development 
(ie. primary, intermediate, & secondary), and one teacher rating schedule across the three 
levels is planned. 

As NEMP’s research last year (Crooks & Flockton, 1998) pointed out students 
generally find information easily but struggle to know how or why to make use of the 
information. Nevertheless, in a battery of assessment instruments it is still useful to 
identify the architecture of a student’s understanding of library systems and their 
purposes and it is toward that goal that these instruments have been developed. 

Instruments 

The tests and rating schedule have been developed through a series of empirical 
trials and reviews (Figure 1). This process has allowed refinement of several dimensions 
of the instruments; the wording of the questions, the layout and ordering of items, the 
validation of the tasks and stimulus material against school library realities, 
determination of the difficulty and discrimination of each item, and the sequencing of the 
information skills curriculum under-girding the instruments. This last dimension has 
been most crucial since detailed expectations of achievement standards do not yet exist 
for information skills. The three levels of the test act as a de-facto statement of the 
minimum content which students could be expected to master. 

The content of the test is broken into four major aspects of library systems that assist 
library users to locate and make use of information (Table 2); the catalogue, library 
sections and shelving, and evaluation of sources.. Some aspects of knowledge are tested 
early on and disappear from the test criteria as children age; for example library sections. 
Some aspects of knowledge (eg. the world wide web and evaluating the authority of 
sources) are introduced later as students develop the reading and information experience 
needed to handle the questions. Other dimensions remain important across all three tests; 
that is knowledge about Dewey decimal system and the use of key words. The 
development or sequencing of these systems is also marked by increasing the amount and 
complexity of information that students have to handle in order to answer the questions. 
The type of response required of students also contributes to increasing difficulty; 
multiple-choice questions are easier in general than constructed responses. To illustrate 
these points a very small sample of items on the use and structure of the Dewey decimal 
system, with results, show an increasingly demanding curriculum expectation (Figure 2). 

. It should be kept in mind that these sequences are valid only for the condition of 
being asked to produce knowledge in a pen & paper, independent, on-demand test 
situation. Obviously, teachers will instruct, model, and support children in understanding 
these content demands much sooner than students could be expected to answer written 
questions independently. Thus, the three tests can indicate to teachers the areas of library 
knowledge students should be able to handle by the end of years 6, 8, and 10 respectively 
if students are to make effective use of the library to meet their own information needs. 

Throughout the design and construction of these instruments emphasis has been put 
on the fact that all library systems, structures, and procedures exist to enable users to 
solve information problems. The kinds of problems users have include: 
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• Does this library have any information pertinent to my problem? 

• What information can 1 get from the catalogue to help solve the problem? 

• How do 1 structure my problem in terms of library systems? 

• Where is the information I want? 

• What form is the information in? 

• What procedures do 1 use when things go wrong (eg. there’s too many 
sources, a source isn’t where it’s supposed to be, and so on)? 

Thus, the tests elicit responses which show that students can both exhibit knowledge 
of the library system itself and knowledge of how to use the library system to meet 
certain information problems. 

In addition to the tests, a teacher rating scale has been in development. This scale 
identifies 1 1 problem situations commonly faced in the process of using catalogues and 
locating information in a library (Table 3). All of the situations, bar knowing how to use 
the catalogue technology, are covered to some degree by test items. Teachers rate 
students on the quality of independence they have observed students exhibiting in the 
problem situations. This scale has the capability of helping teachers identify students 
who may be able to locate information despite their inability to score well on the test. 

Method 

The third trial of the library skills test was carried out by post using whole classes 
selected randomly according to the certain criteria concerning decile, and location (Table 
4). In effect, this provided two classes from four schools for each test with one school 
each at low (1-3) and high (8-10) decile ranges and two schools for the mid (4-7) decile 
range. However, when relying on the good will of others at a distance best laid plans do 
go astray and numbers do not quite match up with the plan. 

Each test is analysed using ConQuest statistical package to determine item 
difficulties and discrimination, as well as overall test reliability. 

In order to validate the teacher rating scale, teachers were asked to rate every second 
student taking the test. Despite this plan going even more astray in terms of data 
collection, sufficient data was captured to carry out a variety of multi-variate statistical 
analyses using SAS. 

Results 

The primary and intermediate tests (Tables 5) are sufficiently reliable to proceed to 
national standardisation with coefficient alpha scores of 0.91 and 0.87 respectively. This 
is quite respectable internal consistency given the low number of items, again 35 and 33 
respectively. Bar some minor modifications, these two tests are sufficiently robust 
technically to proceed to standardisation. The secondary test has a significant number of 
extremely difficult items, especially those that require students to fill out a search field 
accurately (Figure 3), and will require further trialing prior to standardisation. 

Pearson Correlation coefficients (Table 6) indicate a small number of statistically 
significant relationships within the data set. 

• Gender has a positive though weak correlation with all three measurements; 
ie. females tend to do better on the test and tend to be rated higher for 
catalogue and location skills. 

• Test version and year have weak negative correlations with test score and 
catalogue rating. This indicates that catalogue ratings and test scores have a 
small tendency to decrease with age; perhaps suggesting that the tests are 
increasingly harder. 
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• Decile has a weak positive correlation with test score alone implying that 
students from higher decile schools perform better on formal paper and pencil 
testing. 

• The assessment instruments correlate quite well with each other. The test 
score has a moderately strong positive correlation with the two ratings while 
the two ratings are strongly correlated with each other. This implies that 
while the two rating scales are measuring more or less the same characteristic 
as each other they are, as might be expected, measuring a somewhat different 
trait than the test. 

Analysis of means (Tables 7) produces similar results. There is no statistically 
significant difference across the three levels in terms of location rating. Likewise no 
difference exists between the secondary and intermediate tests and ratings results. 
However, the primary test and catalogue rating are significantly different from the other 
two levels. Statistically significant differences exist in favour of the high decile group 
test results only; there are no statistically significant difference for the ratings across all 
decile levels. Perhaps unsurprisingly, differences for the test and both ratings between 
boys and girls are statistically significant in favour of the females. For test scores the 
impact of decile on gender is interesting (Figure 4); girls at low and mid-decile groups 
are very close to each other before stepping up to the high decile girls; whereas boys’ 
scores increase across each decile group. 

Multiple analysis of variance statistics (Tables 8) clearly indicate that the data is 
explained best by the separate main effects of decile, school year, and gender. The test 
scores are driven separately by all three demographic variables; while the two ratings are 
both driven by gender differences only. The only overall significant interaction, which 
explains the data after accounting for the main effects of decile, school year and gender, 
is that of year and decile, in other words scores rise as both the year and the decile rises. 

A principal components analysis (Tabid 9A) clearly shows that after the 
measurement variables, school year, decile, and gender, shape the next 3 descending 
components resulting in four components that explain over 90% of the variance. School 
year and test version are proxies for increasing age of students. A principal factor 
analysis (Table 9B), which reduces the correlation matrix by removing the uniquenesses, 
simplifies the data structure to two factors. The first factor loads heavily on the two 
ratings, moderately on the test score, and weakly on gender. The second factor loads 
heavily on test version and school year or age. 

Conclusions 

This data analysis tells a relatively simple, yet powerful, story about children’s 
library skills. Girls score higher and are rated higher than boys. Students in high decile 
schools receive higher test scores than those at low and mid-decile schools. Older 
students tend to do better than younger ones. And yet, this is not the whole story. The 
fact that the students are not truly randomly sampled, due to the cluster effect of testing 
whole classes, may suggest that what looks like a decile effect may be a school or teacher 
effect that could be replicated in other schools. 

It appears that the tests are moving in the right direction in terms of their ability to 
discriminate reliably among candidates. More importantly, they will permit teachers to 
make informed decisions about the learning needs of a wide range of students. As well 
the teacher rating schedules, while measuring differently from the tests, give an insight 
into student behaviour that coincides, quite strongly for consensual validation, with the 
test scores. It is possible that the rating scales might be successfully used as a substitute 
for the test. 
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Table2: Content Analysis: Library Skills Tests 

Numbers are the question numbers in each test 
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Topic 


Primary 


Intermediate 


Secondary 


Catalogue Use 








Author 


10, 22, 23,27,31,32 


19, 21 


11,23 


Dewey Number 


11, 13, 20, 23,24, 28, 


15, 17, 18,21,22, 23, 


5, 8, 9, 22, 27, 28a, 




29, 30, 34 


24, 26, 27, 28, 29 


28b, 29 


Item Format 




3,8 




Publication Place 




5, 16iii 




Publication Year 


21,22 


6, 16v 


4 


Publisher 




9, 16iv 




Subject / Key Words 


12, 14, 15, 16, 17, 18, 


1, 7, 11, 12, 13, 14 


6, 7, 9, 13, 14, 15, 18, 




20, 26, 29 




20, 22, 24, 26, 29 


Title 


9, 18, 19,21 


16ii 


7,25 


Library Section 








Big Book 


5 






Catalogue 


8 






Fiction 


1,7,33 


2, 20 


8 


Non-Fiction 


3,6, 11,35 


17, 18, 30 




Reference 


2,4 




■1 


Vertical File 




10 




World Wide Web 






19,21 


Library Location 








Search Procedures 


25 


11, 13, 14, 15, 22, 24, 


16, 22,23, 24, 25,26, 






25 


30 


Shelf / Bay 


23,29, 30,31,32, 34 


17, 18, 19, 20, 23,26, 


11, 13,27, 28a, 28b, 






27, 28, 29 


29 


Signposts/Maps 


1,2, 3,4, 5, 6, 7,8 






Spine Labels 


28, 29, 30 




5, 27, 28a, 28b, 29 


Evaluation of Sources 








Authority 






19,21 


Completeness 






2 


Comprehensibility 




4 


3,17 


Currency/Recency 


21,22 


6 


4 


Relevance 






18 









o 
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Table 4: Sampling Design Parameters - Trial 3 Postal Method 



Level 


Primary 


Intermediate 


Secondary 


Year 


5 


6 


7 


8 


9 


10 


Number 


100 


100 


100 


100 


100 


100 



Gender Male 50% Female 50% 



Decile Low{\-7>) Middle {A-1) High{^-\{)) 

25% 50% 25% 



Area Rural/Small Town Provincial Cities Major Urban 
25% 25% 50% 
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Table 5A: Primary Library Skills - Test Results 





Trials 


Trial 2 


Trial 1 


N 


184 


106 


113 


Time 


T4 98 


T3 98 


T2 98 


Year Level 


5 & 6 


5 


6 


# Items 


35 


40 


31 


Average 


19.8 56.7% 


15.0 37.6% 


14.1 45.5% 


Std Dev 


8.01 


7.34 


5.69 


Std Err Avg 


.50 


.71 


.54 


Std Err Meas't 


2.46 


2.73 


2.38 


Coefficient alpha 


.91 


.86 


:82 



Table 5B: Intermediate Library Skills - Test Results 





Trials 


Trial 2 


Trial 1 


N 


237 


124 


100 


Time 


T4 98 


T3 98 


T2 98 


Year Level 


7 & 8 


7 


8 


# Items 


33 


34 


28 


Average 


16.4 49.6% 


15.3 44.9% 


14.6 52.1% 


Std Dev 


6.79 


6.14 


4.59 


Std Err Avg 


.44 


.55 


.46 


Std Err Meas't 


2.47 


2.52 


2.23 


Coefficient alpha 


.87 


.83 


.76 



Table 5C: Secondary Library Skills - Test Results 





Trials 


Trial 2 


Trial 1 


N 


178 


94 


97 


Time 


T4 98 


T3 98 


T2 98 


Year Level 


9 & 10 


9 


10 


# Items 


31 


32 


32 


Average 


17.2 55.3% 


15.2 47.6% 


17.5 54.6% 


Std Dev 


5.56 


5.7 


5.oe 


Std Err Avg 


.42 


.59 


.51 


Std Err Meas't 


2.38 


2.38 


2.33 


Coefficient alpha 


.82 


.83 


.79 
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Table 8 A: Test Score Manova Summary* 



Source 


DF 


SS Type III 


Mean Square 


F Value Pr > F 


Decile 


2 


969.1 


484.5 


11.85 0.0001 


Gender 


1 


1124.7 


1124.7 


27.50 0.0001 


Year 


3 


436.4 


145.5 


3.56 0.0150 


Error 


249 


10183.0 


40.9 




Total 


257 


12866.4 






^ Based on Least Square 


Means, All other effects not significant 


Table 8B: Catalogue Rating Manova Summary* 


Source 


DF 


SS Type III 


Mean Square 


F Value Pr > F 


Gender 


1 


224.4 


224.4 


9.41 0.0024 


Error 


249 


5937.1 


23.8 




Total 


257 


6552.0 






* Based on Least Square 


Means, All other effects not significant 


Table 8C: Location Rating Manova Summary* 


Source 


DF 


SS Type III 


Mean Square 


F Value Pr > F 


Gender 


1 


735.9 


735.9 


19.99 0.0001 


Error 


249 


9167.5 


36.8 




Total 


257 


10279.8 






* Based on Least Square 


Means, All other effects not significant 


Table 8D: Overall Effects Significance* 




Effect 


Wilks ’ Lambda 


F DF 


Pr> F 


Decile 




0.903 


4.308 6 


0.0003 


Year 




0.911 


2.578 9 


0.0064 


Gender 




0.874 


11.773 3 


0.0001 


Year x Decile 


0.894 


1.766 15 


0.0357 



* Based on Least Square Means, All other effects not significant 
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Table 9A: Principal Components Analysis 



Corrtponent 


Proportion 


Members 


Weights 


Meaning 


1 


0.369 


Test 


0.496 


Measurement 






Catalogue 


0.551 








Location 


0.545 




2 


Q211 


Test Version 


0.661 


Age 






School Year 


0.664 




3 


0.141 


Decile Group 


0.921 


Decile 


4 


0.127 


Gender 


0.937 


Gender 


Cumulative 


0.914 









A^= 258, 7 Variables (last 3 components ignored due to small size) 



Table 9B: Principal Factor Analysis 



Factor 


Proportion 


Members 


Weights 


Meaning 


1 


0.563 


Location 


0.909 


Measurement 






Catalogue 


0.890 


& Gender 






Test 


0.687 








Gender 


0.307 




2 


0.442 


Test Version 


0.958 


Age 






School Year 


0.957 
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Figure 1: Design Processes: Library Skills Tests 
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Figure 2: Dewey Decimal Sample Questions 



Primary 



(13) 



[D 



What does the number 523,2 mean? 

(A) Tliere are two books on this 
topic on the shelf, 

(B) There are 523 pages in the 
book, 

(C) There are two books by this 
writer in the library. 

(D) The book is in the 523 
section of the library. 

83% pb 0.49] 



Intermediate 

(17) In which bay on which you would 
find a book with the number 959.9? 

[ D 70% pb 0.391 

(18) Air Transport by Robert Baird at 
number 600 would be in which bay? 

[B 71% pb 0.391 



Secondary 

(28) One book on the shelf is in the wrong 
place. 

(A) Write the full spine label 
number of the one book that 
is in the wrong place. 

[303.3 EHI 54% pb 0.621 

(B) Write the letter (A-F) that 
shows the two books 
between which the mis- 
shelved book should go. 

I A 33% pb 0.44] 





523.2 



Asimov, Isaac 

Our Solar System 

1988 

Gareth Stevens Publishing Company : Milwaukee 
32p,; ill. (some col.); 26 cm. 

Subject: Solar system 
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Figure 3: Some Problematic Items - Secondary Test 

Below is the search screen of a computer library catalogue. Use it to answer 
questions 22 to 25. For each question put your answer in the correct field of the 
catalogue screen. All the answers go in the one screen. 

Write in the correct field exactly what you would type in if you were looking for: 

(22) books on the topic ‘comets - 523.6’. 

(23) a book by Carl Sagan. 

(24) information about “technology used in space”. 

(25) the book The Dragons of Eden. 



Welcome To The School Library Catalogue 

Enter Your Request in the Fields 



Sagan, Carl 21% pbO.27 



Author Search: 

Title Search: ^ Dragons of Eden, The 3% pb 0.10 



Subject/Key Word Search: Space technology 



Dewey Number Search: ( S23.6 75% pb 0.32 



* Remember to check the accuracy of your entry. 



7% pb 0.28 



J 



J 
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Figure 4: Decile & Gender Results 

Library Skills Tests 




1 



2 

Decile Group 
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